Hui, J., 2021. GAN — Wasserstein GAN & WGAN-GP [WWW Document]. Medium. URL https://jonathan-hui.medium.com/gan-wasserstein-gan-wgan-gp-6a1a2aa1b490 (accessed 1.31.23).
Arjovsky, M., Chintala, S., Bottou, L., 2017. Wasserstein GAN [WWW Document]. arXiv.org. URL https://arxiv.org/abs/1701.07875v3 (accessed 2.1.23).
W here is Wasserstein named after mathematical concept wasserstein distance (Arjovsky et al., 2017)
Wasserstein distance is the minimum cost of transporting mass in converting the data distribution q to the data distribution p.

From the WGAN paper (Arjovsky et al., 2017) : The intuition of the formula above is measuring the distance between 2 distribution
Simplified formula: D(x) - D(G(z))
D(x) is the critic's output for a real instance. G(z) is the generator's output when given noise z. D(G(z)) is the critic's output for a fake instance
It takes confidence of the discriminator's prediction on a real image and then deducting the confidennce of the discrinator's prediction for a fake image

From the Wasserstein GAN (Hui, 2021): WGAN first looked at KL and JS divergence and realised that if the 2nd Gaussian distribution Q becomes too large , gradient of divergency diminishes and generator learns nothing.
KL uses JS divergence uses the KL diverge to calculate a more normalized score which gives a value between 0 to 1 . On top of that the measure of the difference between the real and fake distribution is symmetric. People also call the JS divergence the average KL divergence since it multiplies by half of pretty much the KL divergence formula.

The figure above from the original WGAN paper (Arjovsky et al., 2017) shows an optimal discriminator trying to differentiate two Gaussians (aka 2 types of images) using the original loss function Binary Cross entropy = abs(y_pred - y_true). However as it trains , the discriminator draws the boundaries but also result in a vanishing gradient. This is a huge problem because without the critic's gradient being fed to the generator , training process stops. However, the Wasserstein loss appears to provide a clean gradient for the generator to continue learning
# pip install visualkeras
Requirement already satisfied: visualkeras in c:\users\lg\.conda\envs\gpu_env\lib\site-packages (0.0.2)Note: you may need to restart the kernel to use updated packages. Requirement already satisfied: aggdraw>=1.3.11 in c:\users\lg\.conda\envs\gpu_env\lib\site-packages (from visualkeras) (1.3.15) Requirement already satisfied: pillow>=6.2.0 in c:\users\lg\.conda\envs\gpu_env\lib\site-packages (from visualkeras) (9.2.0) Requirement already satisfied: numpy>=1.18.1 in c:\users\lg\.conda\envs\gpu_env\lib\site-packages (from visualkeras) (1.23.4)
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import matplotlib.pyplot as plt
import pandas as pd
import visualkeras
from keras.utils import to_categorical
IMG_SHAPE = (32,32,3)
BATCH_SIZE = 512
# Size of the noise vector
noise_dim = 128
cifar10 = keras.datasets.cifar10
(train_images, train_labels), (test_images, test_labels) = cifar10.load_data()
print(f"Number of examples: {len(train_images)}")
print(f"Shape of the images in the dataset: {train_images.shape[1:]}")
# Reshape each sample to (32,32,3) and normalize the pixel values in the [-1, 1] range
train_images = train_images.reshape(train_images.shape[0], *IMG_SHAPE).astype("float32")
train_images = (train_images - 127.5) / 127.5
Number of examples: 50000 Shape of the images in the dataset: (32, 32, 3)
def conv_block(
x,
filters,
activation,
kernel_size=(3, 3),
strides=(1, 1),
padding="same",
use_bias=True,
use_bn=False,
use_dropout=False,
drop_value=0.5,
):
x = layers.Conv2D(
filters, kernel_size, strides=strides, padding=padding, use_bias=use_bias
)(x)
if use_bn:
x = layers.BatchNormalization()(x)
x = activation(x)
if use_dropout:
x = layers.Dropout(drop_value)(x)
return x
def get_discriminator_model():
img_input = layers.Input(shape=IMG_SHAPE)
# Zero pad the input to make the input images size to (32, 32, 3).
x = layers.ZeroPadding2D((2, 2))(img_input)
x = conv_block(
x,
64,
kernel_size=(5, 5),
strides=(2, 2),
use_bn=False,
use_bias=True,
activation=layers.LeakyReLU(0.2),
use_dropout=False,
drop_value=0.3,
)
x = conv_block(
x,
128,
kernel_size=(5, 5),
strides=(2, 2),
use_bn=False,
activation=layers.LeakyReLU(0.2),
use_bias=True,
use_dropout=True,
drop_value=0.3,
)
x = conv_block(
x,
256,
kernel_size=(5, 5),
strides=(2, 2),
use_bn=False,
activation=layers.LeakyReLU(0.2),
use_bias=True,
use_dropout=True,
drop_value=0.3,
)
x = conv_block(
x,
512,
kernel_size=(5, 5),
strides=(2, 2),
use_bn=False,
activation=layers.LeakyReLU(0.2),
use_bias=True,
use_dropout=False,
drop_value=0.3,
)
x = layers.Flatten()(x)
x = layers.Dropout(0.2)(x)
x = layers.Dense(1)(x)
d_model = keras.models.Model(img_input, x, name="discriminator")
return d_model
d_model = get_discriminator_model()
d_model.summary()
Model: "discriminator"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 32, 32, 3)] 0
zero_padding2d (ZeroPadding (None, 36, 36, 3) 0
2D)
conv2d (Conv2D) (None, 18, 18, 64) 4864
leaky_re_lu (LeakyReLU) (None, 18, 18, 64) 0
conv2d_1 (Conv2D) (None, 9, 9, 128) 204928
leaky_re_lu_1 (LeakyReLU) (None, 9, 9, 128) 0
dropout (Dropout) (None, 9, 9, 128) 0
conv2d_2 (Conv2D) (None, 5, 5, 256) 819456
leaky_re_lu_2 (LeakyReLU) (None, 5, 5, 256) 0
dropout_1 (Dropout) (None, 5, 5, 256) 0
conv2d_3 (Conv2D) (None, 3, 3, 512) 3277312
leaky_re_lu_3 (LeakyReLU) (None, 3, 3, 512) 0
flatten (Flatten) (None, 4608) 0
dropout_2 (Dropout) (None, 4608) 0
dense (Dense) (None, 1) 4609
=================================================================
Total params: 4,311,169
Trainable params: 4,311,169
Non-trainable params: 0
_________________________________________________________________
import visualkeras
visualkeras.layered_view(d_model, legend=True)
def upsample_block(
x,
filters,
activation,
kernel_size=(3, 3),
strides=(1, 1),
up_size=(2, 2),
padding="same",
use_bn=False,
use_bias=True,
use_dropout=False,
drop_value=0.3,
):
x = layers.UpSampling2D(up_size)(x)
x = layers.Conv2D(
filters, kernel_size, strides=strides, padding=padding, use_bias=use_bias
)(x)
if use_bn:
x = layers.BatchNormalization()(x)
if activation:
x = activation(x)
if use_dropout:
x = layers.Dropout(drop_value)(x)
return x
def get_generator_model():
noise = layers.Input(shape=(noise_dim,))
x = layers.Dense(4 * 4 * 256, use_bias=False)(noise)
x = layers.BatchNormalization()(x)
x = layers.LeakyReLU(0.2)(x)
x = layers.Reshape((4, 4, 256))(x)
x = upsample_block(
x,
128,
layers.LeakyReLU(0.2),
strides=(1, 1),
use_bias=False,
use_bn=True,
padding="same",
use_dropout=False,
)
x = upsample_block(
x,
64,
layers.LeakyReLU(0.2),
strides=(1, 1),
use_bias=False,
use_bn=True,
padding="same",
use_dropout=False,
)
x = upsample_block(
x, 3, layers.Activation("tanh"), strides=(1, 1), use_bias=False, use_bn=True
)
# At this point, we have an output which has the same shape as the input, (32, 32, 1).
# We will use a Cropping2D layer to make it (28, 28, 1).
# x = layers.Cropping2D((2, 2))(x)
# remove cropping layer?
g_model = keras.models.Model(noise, x, name="generator")
return g_model
g_model = get_generator_model()
g_model.summary()
Model: "generator"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 128)] 0
dense_1 (Dense) (None, 4096) 524288
batch_normalization (BatchN (None, 4096) 16384
ormalization)
leaky_re_lu_4 (LeakyReLU) (None, 4096) 0
reshape (Reshape) (None, 4, 4, 256) 0
up_sampling2d (UpSampling2D (None, 8, 8, 256) 0
)
conv2d_4 (Conv2D) (None, 8, 8, 128) 294912
batch_normalization_1 (Batc (None, 8, 8, 128) 512
hNormalization)
leaky_re_lu_5 (LeakyReLU) (None, 8, 8, 128) 0
up_sampling2d_1 (UpSampling (None, 16, 16, 128) 0
2D)
conv2d_5 (Conv2D) (None, 16, 16, 64) 73728
batch_normalization_2 (Batc (None, 16, 16, 64) 256
hNormalization)
leaky_re_lu_6 (LeakyReLU) (None, 16, 16, 64) 0
up_sampling2d_2 (UpSampling (None, 32, 32, 64) 0
2D)
conv2d_6 (Conv2D) (None, 32, 32, 3) 1728
batch_normalization_3 (Batc (None, 32, 32, 3) 12
hNormalization)
activation (Activation) (None, 32, 32, 3) 0
=================================================================
Total params: 911,820
Trainable params: 903,238
Non-trainable params: 8,582
_________________________________________________________________
visualkeras.layered_view(g_model, legend=True)
class WGAN(keras.Model):
def __init__(
self,
discriminator,
generator,
latent_dim,
discriminator_extra_steps=3,
gp_weight=10.0,
):
super().__init__()
self.discriminator = discriminator
self.generator = generator
self.latent_dim = latent_dim
self.d_steps = discriminator_extra_steps
self.gp_weight = gp_weight
def compile(self, d_optimizer, g_optimizer, d_loss_fn, g_loss_fn):
super().compile()
self.d_optimizer = d_optimizer
self.g_optimizer = g_optimizer
self.d_loss_fn = d_loss_fn
self.g_loss_fn = g_loss_fn
def gradient_penalty(self, batch_size, real_images, fake_images):
"""Calculates the gradient penalty.
This loss is calculated on an interpolated image
and added to the discriminator loss.
"""
# Get the interpolated image
alpha = tf.random.normal([batch_size, 1, 1, 1], 0.0, 1.0)
diff = fake_images - real_images
interpolated = real_images + alpha * diff
with tf.GradientTape() as gp_tape:
gp_tape.watch(interpolated)
# 1. Get the discriminator output for this interpolated image.
pred = self.discriminator(interpolated, training=True)
# 2. Calculate the gradients w.r.t to this interpolated image.
grads = gp_tape.gradient(pred, [interpolated])[0]
# 3. Calculate the norm of the gradients.
norm = tf.sqrt(tf.reduce_sum(tf.square(grads), axis=[1, 2, 3]))
gp = tf.reduce_mean((norm - 1.0) ** 2)
return gp
def train_step(self, real_images):
if isinstance(real_images, tuple):
real_images = real_images[0]
# Get the batch size
batch_size = tf.shape(real_images)[0]
# For each batch, we are going to perform the
# following steps as laid out in the original paper:
# 1. Train the generator and get the generator loss
# 2. Train the discriminator and get the discriminator loss
# 3. Calculate the gradient penalty
# 4. Multiply this gradient penalty with a constant weight factor
# 5. Add the gradient penalty to the discriminator loss
# 6. Return the generator and discriminator losses as a loss dictionary
# Train the discriminator first. The original paper recommends training
# the discriminator for `x` more steps (typically 5) as compared to
# one step of the generator. Here we will train it for 3 extra steps
# as compared to 5 to reduce the training time.
for i in range(self.d_steps):
# Get the latent vector
random_latent_vectors = tf.random.normal(
shape=(batch_size, self.latent_dim)
)
with tf.GradientTape() as tape:
# Generate fake images from the latent vector
fake_images = self.generator(random_latent_vectors, training=True)
# Get the logits for the fake images
fake_logits = self.discriminator(fake_images, training=True)
# Get the logits for the real images
real_logits = self.discriminator(real_images, training=True)
# Calculate the discriminator loss using the fake and real image logits
d_cost = self.d_loss_fn(real_img=real_logits, fake_img=fake_logits)
# Calculate the gradient penalty
gp = self.gradient_penalty(batch_size, real_images, fake_images)
# Add the gradient penalty to the original discriminator loss
d_loss = d_cost + gp * self.gp_weight
# Get the gradients w.r.t the discriminator loss
d_gradient = tape.gradient(d_loss, self.discriminator.trainable_variables)
# Update the weights of the discriminator using the discriminator optimizer
self.d_optimizer.apply_gradients(
zip(d_gradient, self.discriminator.trainable_variables)
)
# Train the generator
# Get the latent vector
random_latent_vectors = tf.random.normal(shape=(batch_size, self.latent_dim))
with tf.GradientTape() as tape:
# Generate fake images using the generator
generated_images = self.generator(random_latent_vectors, training=True)
# Get the discriminator logits for fake images
gen_img_logits = self.discriminator(generated_images, training=True)
# Calculate the generator loss
g_loss = self.g_loss_fn(gen_img_logits)
# Get the gradients w.r.t the generator loss
gen_gradient = tape.gradient(g_loss, self.generator.trainable_variables)
# Update the weights of the generator using the generator optimizer
self.g_optimizer.apply_gradients(
zip(gen_gradient, self.generator.trainable_variables)
)
return {"d_loss": d_loss, "g_loss": g_loss}
class GANMonitor(keras.callbacks.Callback):
def __init__(self, num_img=100, latent_dim=128):
self.num_img = num_img
self.latent_dim = latent_dim
self.vmin = 0
self.vmax = 1
def on_epoch_end(self, epoch, logs=None):
random_latent_vectors = tf.random.normal(shape=(self.num_img, self.latent_dim))
generated_images = self.model.generator(random_latent_vectors)
generated_images -= self.vmin
generated_images /= (self.vmax - self.vmin)
# Create a figure with 10 rows and 10 columns
if (epoch % 50 == 1):
fig, axes = plt.subplots(10,10, figsize=(10, 10))
axes = axes.ravel()
for i in range(self.num_img):
img = keras.preprocessing.image.array_to_img(generated_images[i])
axes[i].imshow(img)
axes[i].axis("off")
title = "Epoch " + str(epoch)
plt.suptitle(title)
if (epoch % 250 == 1):
path = "./generated_img_" + title
plt.savefig(path,dpi=300)
plt.show()
# Instantiate the optimizer for both networks
# (learning_rate=0.0002, beta_1=0.5 are recommended)
generator_optimizer = keras.optimizers.Adam(
learning_rate=0.0002, beta_1=0.5, beta_2=0.9
)
discriminator_optimizer = keras.optimizers.Adam(
learning_rate=0.0002, beta_1=0.5, beta_2=0.9
)
# Define the loss functions for the discriminator,
# which should be (fake_loss - real_loss).
# We will add the gradient penalty later to this loss function.
def discriminator_loss(real_img, fake_img):
real_loss = tf.reduce_mean(real_img)
fake_loss = tf.reduce_mean(fake_img)
return fake_loss - real_loss
# Define the loss functions for the generator.
def generator_loss(fake_img):
return -tf.reduce_mean(fake_img)
# Set the number of epochs for trainining.
epochs = 300
# Instantiate the customer `GANMonitor` Keras callback.
cbk = GANMonitor(num_img=100, latent_dim=noise_dim)
# Get the wgan model
wgan = WGAN(
discriminator=d_model,
generator=g_model,
latent_dim=noise_dim,
discriminator_extra_steps=3,
)
# Compile the wgan model
wgan.compile(
d_optimizer=discriminator_optimizer,
g_optimizer=generator_optimizer,
g_loss_fn=generator_loss,
d_loss_fn=discriminator_loss,
)
# Start training
hist = wgan.fit(train_images, batch_size=BATCH_SIZE, epochs=epochs, callbacks=[cbk])
Epoch 1/300 98/98 [==============================] - 57s 493ms/step - d_loss: -7.3406 - g_loss: 21.8229 Epoch 2/300 98/98 [==============================] - ETA: 0s - d_loss: -4.1790 - g_loss: 12.7049
98/98 [==============================] - 50s 514ms/step - d_loss: -4.1810 - g_loss: 12.6020 Epoch 3/300 98/98 [==============================] - 48s 486ms/step - d_loss: -3.7075 - g_loss: 5.0171 Epoch 4/300 98/98 [==============================] - 48s 487ms/step - d_loss: -3.4194 - g_loss: 1.5335 Epoch 5/300 98/98 [==============================] - 48s 488ms/step - d_loss: -3.0173 - g_loss: -6.4131 Epoch 6/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.9612 - g_loss: -1.9092 Epoch 7/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.9240 - g_loss: -1.0536 Epoch 8/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.8378 - g_loss: -5.3020 Epoch 9/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.8439 - g_loss: -2.2552 Epoch 10/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.7490 - g_loss: -3.3330 Epoch 11/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.7314 - g_loss: -3.8568 Epoch 12/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.6351 - g_loss: -7.1157 Epoch 13/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.5093 - g_loss: -8.9468 Epoch 14/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.4658 - g_loss: -4.1055 Epoch 15/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.1847 - g_loss: -8.2322 Epoch 16/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.2778 - g_loss: -9.5953 Epoch 17/300 98/98 [==============================] - 48s 488ms/step - d_loss: -2.3049 - g_loss: -8.6796 Epoch 18/300 98/98 [==============================] - 48s 492ms/step - d_loss: -2.2723 - g_loss: -10.8831 Epoch 19/300 98/98 [==============================] - 47s 480ms/step - d_loss: -1.9372 - g_loss: -7.9568 Epoch 20/300 98/98 [==============================] - 47s 483ms/step - d_loss: -1.9333 - g_loss: -6.8688 Epoch 21/300 98/98 [==============================] - 47s 480ms/step - d_loss: -1.9050 - g_loss: -1.9870 Epoch 22/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.7534 - g_loss: -4.2289 Epoch 23/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.7087 - g_loss: 4.0505 Epoch 24/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.8705 - g_loss: -1.0945 Epoch 25/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.7655 - g_loss: -1.0792 Epoch 26/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.8105 - g_loss: -0.6885 Epoch 27/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.8894 - g_loss: 0.1071 Epoch 28/300 98/98 [==============================] - 47s 478ms/step - d_loss: -1.7708 - g_loss: -0.8322 Epoch 29/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.7459 - g_loss: -1.0736 Epoch 30/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.6770 - g_loss: 1.0342 Epoch 31/300 98/98 [==============================] - 47s 478ms/step - d_loss: -1.7013 - g_loss: 0.0665 Epoch 32/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.6664 - g_loss: -3.6770 Epoch 33/300 98/98 [==============================] - 47s 479ms/step - d_loss: -1.7129 - g_loss: -0.9532 Epoch 34/300 98/98 [==============================] - 48s 492ms/step - d_loss: -1.6525 - g_loss: 2.0594 Epoch 35/300 98/98 [==============================] - 48s 487ms/step - d_loss: -1.5863 - g_loss: -2.5276 Epoch 36/300 98/98 [==============================] - 48s 487ms/step - d_loss: -1.6913 - g_loss: -1.3285 Epoch 37/300 98/98 [==============================] - 48s 487ms/step - d_loss: -1.6211 - g_loss: -2.1618 Epoch 38/300 98/98 [==============================] - 47s 481ms/step - d_loss: -1.6025 - g_loss: 1.6889 Epoch 39/300 98/98 [==============================] - 47s 477ms/step - d_loss: -1.6253 - g_loss: 0.6202 Epoch 40/300 98/98 [==============================] - 47s 477ms/step - d_loss: -1.6837 - g_loss: 0.0393 Epoch 41/300 98/98 [==============================] - 47s 477ms/step - d_loss: -1.6113 - g_loss: -4.3944 Epoch 42/300 98/98 [==============================] - 48s 485ms/step - d_loss: -1.5834 - g_loss: -1.0026 Epoch 43/300 98/98 [==============================] - 48s 487ms/step - d_loss: -1.5120 - g_loss: 1.5168 Epoch 44/300 98/98 [==============================] - 48s 487ms/step - d_loss: -1.6232 - g_loss: 2.1421 Epoch 45/300 98/98 [==============================] - 48s 488ms/step - d_loss: -1.5178 - g_loss: 0.2262 Epoch 46/300 98/98 [==============================] - 47s 478ms/step - d_loss: -1.4970 - g_loss: -1.3411 Epoch 47/300 98/98 [==============================] - 45s 462ms/step - d_loss: -1.5803 - g_loss: -1.3414 Epoch 48/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.6683 - g_loss: -2.5523 Epoch 49/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.5444 - g_loss: 0.0131 Epoch 50/300 98/98 [==============================] - 45s 455ms/step - d_loss: -1.5371 - g_loss: -1.2098 Epoch 51/300 98/98 [==============================] - 45s 455ms/step - d_loss: -1.6387 - g_loss: 0.4212 Epoch 52/300 98/98 [==============================] - ETA: 0s - d_loss: -1.5250 - g_loss: 0.2688
98/98 [==============================] - 47s 478ms/step - d_loss: -1.5237 - g_loss: 0.2351 Epoch 53/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.5084 - g_loss: -0.9236 Epoch 54/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.5153 - g_loss: -1.2109 Epoch 55/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.5129 - g_loss: 0.2403 Epoch 56/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.5605 - g_loss: 1.3756 Epoch 57/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.4074 - g_loss: -1.5645 Epoch 58/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.4712 - g_loss: -2.6132 Epoch 59/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.4775 - g_loss: 3.8631 Epoch 60/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.5548 - g_loss: 0.8922 Epoch 61/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.4649 - g_loss: -4.5824 Epoch 62/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.5285 - g_loss: -2.4317 Epoch 63/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.4304 - g_loss: -2.4881 Epoch 64/300 98/98 [==============================] - 45s 456ms/step - d_loss: -1.4473 - g_loss: -1.1217 Epoch 65/300 98/98 [==============================] - 45s 461ms/step - d_loss: -1.5040 - g_loss: 2.7232 Epoch 66/300 98/98 [==============================] - 45s 461ms/step - d_loss: -1.4376 - g_loss: -3.5250 Epoch 67/300 98/98 [==============================] - 45s 457ms/step - d_loss: -1.4372 - g_loss: 1.2274 Epoch 68/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.4734 - g_loss: -1.7833 Epoch 69/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.4026 - g_loss: -2.7036 Epoch 70/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3780 - g_loss: -0.3760 Epoch 71/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3971 - g_loss: 0.2833 Epoch 72/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.4545 - g_loss: 0.6909 Epoch 73/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3889 - g_loss: 2.4488 Epoch 74/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3800 - g_loss: 0.9237 Epoch 75/300 98/98 [==============================] - 45s 457ms/step - d_loss: -1.4084 - g_loss: 0.4737 Epoch 76/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.4056 - g_loss: 0.7281 Epoch 77/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3699 - g_loss: 1.4609 Epoch 78/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.4318 - g_loss: -2.5051 Epoch 79/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3597 - g_loss: -0.8964 Epoch 80/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3212 - g_loss: 0.8537 Epoch 81/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3717 - g_loss: 2.4114 Epoch 82/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3176 - g_loss: -1.2447 Epoch 83/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3894 - g_loss: 1.0681 Epoch 84/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3219 - g_loss: 1.3581 Epoch 85/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3531 - g_loss: 1.3054 Epoch 86/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.2949 - g_loss: -0.5320 Epoch 87/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3341 - g_loss: 0.9290 Epoch 88/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3099 - g_loss: 1.3935 Epoch 89/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.2745 - g_loss: 0.4241 Epoch 90/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3496 - g_loss: 1.5486 Epoch 91/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.2761 - g_loss: 1.3240 Epoch 92/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3818 - g_loss: 1.5650 Epoch 93/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3010 - g_loss: 1.3619 Epoch 94/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3310 - g_loss: 1.3199 Epoch 95/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.3286 - g_loss: 4.0010 Epoch 96/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.2649 - g_loss: 1.1841 Epoch 97/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.3311 - g_loss: 0.9100 Epoch 98/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3023 - g_loss: 0.6819 Epoch 99/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3255 - g_loss: 0.5858 Epoch 100/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.3475 - g_loss: 2.6210 Epoch 101/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.2194 - g_loss: 3.1946 Epoch 102/300 98/98 [==============================] - ETA: 0s - d_loss: -1.3056 - g_loss: 1.8696
98/98 [==============================] - 47s 482ms/step - d_loss: -1.3120 - g_loss: 1.8494 Epoch 103/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.2848 - g_loss: 1.4237 Epoch 104/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.3256 - g_loss: 1.1203 Epoch 105/300 98/98 [==============================] - 45s 455ms/step - d_loss: -1.2511 - g_loss: 1.0639 Epoch 106/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2916 - g_loss: -0.7054 Epoch 107/300 98/98 [==============================] - 45s 455ms/step - d_loss: -1.1969 - g_loss: 0.4681 Epoch 108/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.2933 - g_loss: 1.3242 Epoch 109/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.2012 - g_loss: 2.2679 Epoch 110/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.3089 - g_loss: 0.6502 Epoch 111/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.1943 - g_loss: 1.5444 Epoch 112/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.2381 - g_loss: 2.6213 Epoch 113/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.2571 - g_loss: 3.5570 Epoch 114/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2228 - g_loss: 1.4997 Epoch 115/300 98/98 [==============================] - 45s 455ms/step - d_loss: -1.2060 - g_loss: 2.0576 Epoch 116/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.2086 - g_loss: 1.5745 Epoch 117/300 98/98 [==============================] - 45s 455ms/step - d_loss: -1.1207 - g_loss: 2.1902 Epoch 118/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2450 - g_loss: 2.0470 Epoch 119/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2803 - g_loss: 1.1288 Epoch 120/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2472 - g_loss: 4.5549 Epoch 121/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2029 - g_loss: 3.3712 Epoch 122/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2363 - g_loss: 1.9598 Epoch 123/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1100 - g_loss: 1.3313 Epoch 124/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2277 - g_loss: 1.6718 Epoch 125/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2280 - g_loss: 0.8758 Epoch 126/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1831 - g_loss: 2.6083 Epoch 127/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.2510 - g_loss: 2.0048 Epoch 128/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.2414 - g_loss: 2.6791 Epoch 129/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2265 - g_loss: 1.6053 Epoch 130/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2032 - g_loss: 1.5942 Epoch 131/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1996 - g_loss: 0.3605 Epoch 132/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.1722 - g_loss: 1.2500 Epoch 133/300 98/98 [==============================] - 45s 458ms/step - d_loss: -1.2058 - g_loss: 2.9096 Epoch 134/300 98/98 [==============================] - 45s 458ms/step - d_loss: -1.2217 - g_loss: 2.6260 Epoch 135/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1867 - g_loss: 2.9725 Epoch 136/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.2068 - g_loss: 3.9829 Epoch 137/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1420 - g_loss: 3.3342 Epoch 138/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1942 - g_loss: 2.5772 Epoch 139/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1858 - g_loss: 4.1331 Epoch 140/300 98/98 [==============================] - 46s 465ms/step - d_loss: -1.1805 - g_loss: 1.7865 Epoch 141/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1272 - g_loss: 1.8605 Epoch 142/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1219 - g_loss: 1.3973 Epoch 143/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.0834 - g_loss: 0.9771 Epoch 144/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1768 - g_loss: 1.1102 Epoch 145/300 98/98 [==============================] - 45s 457ms/step - d_loss: -1.1459 - g_loss: 1.1065 Epoch 146/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1261 - g_loss: 1.6852 Epoch 147/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1674 - g_loss: 2.6545 Epoch 148/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0942 - g_loss: 1.8003 Epoch 149/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1106 - g_loss: 3.0615 Epoch 150/300 98/98 [==============================] - 45s 456ms/step - d_loss: -1.1636 - g_loss: 5.3379 Epoch 151/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1802 - g_loss: 3.7367 Epoch 152/300 98/98 [==============================] - ETA: 0s - d_loss: -1.1578 - g_loss: 2.6614
98/98 [==============================] - 48s 488ms/step - d_loss: -1.1581 - g_loss: 2.6190 Epoch 153/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1354 - g_loss: 2.7578 Epoch 154/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1498 - g_loss: 3.7828 Epoch 155/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1098 - g_loss: 3.3301 Epoch 156/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.1638 - g_loss: 2.1495 Epoch 157/300 98/98 [==============================] - 45s 464ms/step - d_loss: -1.1111 - g_loss: 0.8604 Epoch 158/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.2022 - g_loss: 1.7162 Epoch 159/300 98/98 [==============================] - 46s 465ms/step - d_loss: -1.1373 - g_loss: 2.4267 Epoch 160/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1175 - g_loss: 2.6139 Epoch 161/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1568 - g_loss: 3.1018 Epoch 162/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1616 - g_loss: 0.4112 Epoch 163/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.2065 - g_loss: 2.1675 Epoch 164/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1737 - g_loss: 3.2349 Epoch 165/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.0799 - g_loss: 0.6955 Epoch 166/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.1050 - g_loss: -0.3816 Epoch 167/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.0218 - g_loss: 1.3857 Epoch 168/300 98/98 [==============================] - 46s 466ms/step - d_loss: -1.0829 - g_loss: 2.3473 Epoch 169/300 98/98 [==============================] - 45s 454ms/step - d_loss: -1.0777 - g_loss: 2.8695 Epoch 170/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0733 - g_loss: 2.7384 Epoch 171/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0903 - g_loss: 2.4654 Epoch 172/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0632 - g_loss: 2.6664 Epoch 173/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0899 - g_loss: 4.2382 Epoch 174/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0500 - g_loss: 1.8510 Epoch 175/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0774 - g_loss: 3.3243 Epoch 176/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1107 - g_loss: 3.3604 Epoch 177/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1003 - g_loss: 1.8708 Epoch 178/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1401 - g_loss: 2.2696 Epoch 179/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0435 - g_loss: 0.2291 Epoch 180/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0282 - g_loss: 2.5060 Epoch 181/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1153 - g_loss: 2.8579 Epoch 182/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1233 - g_loss: 1.3153 Epoch 183/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0882 - g_loss: 2.2274 Epoch 184/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1000 - g_loss: 1.7496 Epoch 185/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1021 - g_loss: 2.4306 Epoch 186/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0370 - g_loss: 2.0342 Epoch 187/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0428 - g_loss: 2.8306 Epoch 188/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1008 - g_loss: 2.4283 Epoch 189/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1217 - g_loss: 2.7710 Epoch 190/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0630 - g_loss: 2.5978 Epoch 191/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0296 - g_loss: 3.2917 Epoch 192/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.0066 - g_loss: 2.3757 Epoch 193/300 98/98 [==============================] - 45s 456ms/step - d_loss: -1.0698 - g_loss: 0.1348 Epoch 194/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0121 - g_loss: 1.0264 Epoch 195/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9906 - g_loss: 1.3570 Epoch 196/300 98/98 [==============================] - 44s 454ms/step - d_loss: -1.0438 - g_loss: 2.2373 Epoch 197/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0402 - g_loss: -0.1310 Epoch 198/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0488 - g_loss: 3.4424 Epoch 199/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.1399 - g_loss: 4.2830 Epoch 200/300 98/98 [==============================] - 45s 459ms/step - d_loss: -1.0664 - g_loss: 3.6309 Epoch 201/300 98/98 [==============================] - 45s 460ms/step - d_loss: -0.9763 - g_loss: 4.2395 Epoch 202/300 98/98 [==============================] - ETA: 0s - d_loss: -1.0156 - g_loss: 3.0092
98/98 [==============================] - 47s 480ms/step - d_loss: -1.0251 - g_loss: 2.9691 Epoch 203/300 98/98 [==============================] - 44s 452ms/step - d_loss: -1.0047 - g_loss: 2.4707 Epoch 204/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0519 - g_loss: 2.5521 Epoch 205/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9330 - g_loss: 1.7011 Epoch 206/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0218 - g_loss: 1.6597 Epoch 207/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9699 - g_loss: 1.5330 Epoch 208/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0143 - g_loss: 1.2662 Epoch 209/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9972 - g_loss: 0.4574 Epoch 210/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0503 - g_loss: 0.5067 Epoch 211/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9710 - g_loss: 1.3070 Epoch 212/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0635 - g_loss: 0.3515 Epoch 213/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9929 - g_loss: 0.0164 Epoch 214/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0498 - g_loss: 1.6477 Epoch 215/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9700 - g_loss: 0.7667 Epoch 216/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0267 - g_loss: -0.8543 Epoch 217/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9466 - g_loss: 1.1950 Epoch 218/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0177 - g_loss: 0.7497 Epoch 219/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9565 - g_loss: 1.1094 Epoch 220/300 98/98 [==============================] - 45s 457ms/step - d_loss: -1.0675 - g_loss: 2.2064 Epoch 221/300 98/98 [==============================] - 45s 459ms/step - d_loss: -1.0404 - g_loss: 0.8887 Epoch 222/300 98/98 [==============================] - 45s 459ms/step - d_loss: -0.9818 - g_loss: 1.1712 Epoch 223/300 98/98 [==============================] - 45s 459ms/step - d_loss: -1.0265 - g_loss: 2.2391 Epoch 224/300 98/98 [==============================] - 45s 459ms/step - d_loss: -0.8715 - g_loss: 3.4849 Epoch 225/300 98/98 [==============================] - 45s 459ms/step - d_loss: -1.0107 - g_loss: 1.8880 Epoch 226/300 98/98 [==============================] - 45s 459ms/step - d_loss: -1.0036 - g_loss: 1.7467 Epoch 227/300 98/98 [==============================] - 45s 459ms/step - d_loss: -0.9965 - g_loss: 2.6096 Epoch 228/300 98/98 [==============================] - 45s 460ms/step - d_loss: -0.9812 - g_loss: 1.6501 Epoch 229/300 98/98 [==============================] - 45s 459ms/step - d_loss: -0.9855 - g_loss: 2.3651 Epoch 230/300 98/98 [==============================] - 45s 459ms/step - d_loss: -1.0264 - g_loss: 3.1975 Epoch 231/300 98/98 [==============================] - 45s 460ms/step - d_loss: -1.0250 - g_loss: 0.7918 Epoch 232/300 98/98 [==============================] - 45s 460ms/step - d_loss: -0.9574 - g_loss: 0.7262 Epoch 233/300 98/98 [==============================] - 45s 457ms/step - d_loss: -0.9317 - g_loss: -1.0323 Epoch 234/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9810 - g_loss: -0.4640 Epoch 235/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9978 - g_loss: 0.6541 Epoch 236/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0407 - g_loss: 1.0960 Epoch 237/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9414 - g_loss: 1.9438 Epoch 238/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9657 - g_loss: -0.6193 Epoch 239/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9495 - g_loss: 1.5019 Epoch 240/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9005 - g_loss: 2.8085 Epoch 241/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.8969 - g_loss: 1.0215 Epoch 242/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9676 - g_loss: 1.0533 Epoch 243/300 98/98 [==============================] - 44s 453ms/step - d_loss: -1.0075 - g_loss: 1.4819 Epoch 244/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9642 - g_loss: 2.1879 Epoch 245/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9639 - g_loss: 0.7668 Epoch 246/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9948 - g_loss: -0.5589 Epoch 247/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9146 - g_loss: 2.3022 Epoch 248/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9961 - g_loss: 0.5395 Epoch 249/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9905 - g_loss: 0.5777 Epoch 250/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.8771 - g_loss: 0.2384 Epoch 251/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9145 - g_loss: 0.8778 Epoch 252/300 98/98 [==============================] - ETA: 0s - d_loss: -0.9677 - g_loss: 1.6473
98/98 [==============================] - 47s 483ms/step - d_loss: -0.9694 - g_loss: 1.6683 Epoch 253/300 98/98 [==============================] - 44s 452ms/step - d_loss: -0.9292 - g_loss: 0.7109 Epoch 254/300 98/98 [==============================] - 44s 452ms/step - d_loss: -0.9489 - g_loss: 1.2075 Epoch 255/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.8722 - g_loss: 1.7124 Epoch 256/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9398 - g_loss: 0.4124 Epoch 257/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9371 - g_loss: 2.0218 Epoch 258/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9235 - g_loss: 1.3930 Epoch 259/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9945 - g_loss: 2.2995 Epoch 260/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9156 - g_loss: 2.4820 Epoch 261/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9120 - g_loss: 2.6750 Epoch 262/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9213 - g_loss: 2.3087 Epoch 263/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.8734 - g_loss: 1.2678 Epoch 264/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9443 - g_loss: 1.5327 Epoch 265/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9587 - g_loss: 2.6723 Epoch 266/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9363 - g_loss: 2.5637 Epoch 267/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9374 - g_loss: 1.9842 Epoch 268/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.8912 - g_loss: 1.9781 Epoch 269/300 98/98 [==============================] - 44s 453ms/step - d_loss: -0.9396 - g_loss: 1.4545 Epoch 270/300 98/98 [==============================] - 47s 477ms/step - d_loss: -0.9179 - g_loss: 0.9802 Epoch 271/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.9837 - g_loss: 0.6298 Epoch 272/300 98/98 [==============================] - 48s 485ms/step - d_loss: -0.9375 - g_loss: 0.2567 Epoch 273/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.9702 - g_loss: 0.6118 Epoch 274/300 98/98 [==============================] - 47s 484ms/step - d_loss: -1.0137 - g_loss: 0.4975 Epoch 275/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.9480 - g_loss: -0.3920 Epoch 276/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.8967 - g_loss: 1.1590 Epoch 277/300 98/98 [==============================] - 47s 476ms/step - d_loss: -0.8559 - g_loss: 1.7176 Epoch 278/300 98/98 [==============================] - 44s 452ms/step - d_loss: -0.8888 - g_loss: 1.5713 Epoch 279/300 98/98 [==============================] - 44s 452ms/step - d_loss: -0.9089 - g_loss: -0.1344 Epoch 280/300 98/98 [==============================] - 46s 470ms/step - d_loss: -0.9031 - g_loss: -0.1420 Epoch 281/300 98/98 [==============================] - 47s 481ms/step - d_loss: -0.8853 - g_loss: -0.2632 Epoch 282/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.9200 - g_loss: 1.7686 Epoch 283/300 98/98 [==============================] - 47s 481ms/step - d_loss: -0.8523 - g_loss: 1.4383 Epoch 284/300 98/98 [==============================] - 47s 481ms/step - d_loss: -0.9040 - g_loss: 1.0951 Epoch 285/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.9394 - g_loss: 2.6546 Epoch 286/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.9170 - g_loss: 0.1600 Epoch 287/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.9561 - g_loss: -0.1924 Epoch 288/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.8706 - g_loss: 1.3418 Epoch 289/300 98/98 [==============================] - 47s 481ms/step - d_loss: -0.9435 - g_loss: 1.6790 Epoch 290/300 98/98 [==============================] - 47s 484ms/step - d_loss: -0.9301 - g_loss: 1.1571 Epoch 291/300 98/98 [==============================] - 47s 481ms/step - d_loss: -0.8730 - g_loss: 1.8544 Epoch 292/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.8819 - g_loss: 1.2300 Epoch 293/300 98/98 [==============================] - 47s 483ms/step - d_loss: -0.8592 - g_loss: 1.3509 Epoch 294/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.9590 - g_loss: 0.9401 Epoch 295/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.9202 - g_loss: 2.5157 Epoch 296/300 98/98 [==============================] - 48s 485ms/step - d_loss: -0.9116 - g_loss: 2.0525 Epoch 297/300 98/98 [==============================] - 47s 481ms/step - d_loss: -0.8655 - g_loss: 0.4395 Epoch 298/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.9548 - g_loss: 2.3430 Epoch 299/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.9313 - g_loss: 2.3673 Epoch 300/300 98/98 [==============================] - 47s 482ms/step - d_loss: -0.8926 - g_loss: 1.3399
# story history object into dataframe
hist_df = pd.DataFrame(hist.history)
# using pandas dataframe to plot out learning curve
with plt.style.context('seaborn'):
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 8), tight_layout=True)
hist_df.loc[:, ["d_loss", 'g_loss']].plot(ax=ax1, title=r'Learning Curve of Loss Function CE')
plt.show()
Test was smooth and loss was very stable even for high epochs
tf.keras.models.save_model(d_model , "./discriminator_wgan4.h5")
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
tf.keras.models.save_model(g_model , "./generator_wgan4.h5")
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
gen4=tf.keras.models.load_model("./Models/WGAN/generator_wgan4.h5")
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
random_latent_vectors = tf.random.normal(shape=(100, 128))
generated_images =gen4(random_latent_vectors)
generated_images -= -1
generated_images /= (1 - 0)
# Create a figure with 10 rows and 10 column
fig, axes = plt.subplots(10,10, figsize=(10, 10))
axes = axes.ravel()
for i in range(100):
img = keras.preprocessing.image.array_to_img(generated_images[i])
axes[i].imshow(img)
axes[i].axis("off")
# import relevant libraries to calculate FID
import numpy as np
from tensorflow.keras.applications.inception_v3 import InceptionV3, preprocess_input
from tensorflow.image import resize
from scipy.linalg import sqrtm
import math
from tqdm import tqdm
import tensorflow as tf
class GAN_FID:
def __init__(self, batch_size, latent_dim, sample_size, buffer_size):
# setting Hyperparameters
self.BATCH_SIZE = batch_size
self.LATENT_DIM = latent_dim
self.SAMPLE_SIZE = sample_size
self.BUFFER_SIZE = buffer_size
# setting Constants
self.INCEPTION_SHAPE = (299, 299, 3)
self.INCEPTION = InceptionV3(include_top=False, pooling='avg', input_shape=self.INCEPTION_SHAPE)
self.AUTO = tf.data.AUTOTUNE
# method to set generator and training data
def fit(self, generator, train_data):
# setting generative model and original data used for training
self.GENERATOR = generator
self.train_data = train_data
# Preparing Real Images
trainloader = tf.data.Dataset.from_tensor_slices((self.train_data))
trainloader = (
trainloader
.shuffle(self.BUFFER_SIZE)
.map(self.__resize_and_preprocess, num_parallel_calls=self.AUTO)
.batch(self.BATCH_SIZE, num_parallel_calls=self.AUTO)
.prefetch(self.AUTO)
)
self.trainloader = trainloader
# Generate and prepare Synthetic Images (Fake)
rand_labels = np.random.randint(low=0, high=10, size=self.SAMPLE_SIZE)
rand_labels = to_categorical(rand_labels)
random_latent_vectors = tf.random.normal(shape=(1000, self.LATENT_DIM))
generated_images = generator(random_latent_vectors)
genloader = tf.data.Dataset.from_tensor_slices(generated_images)
genloader = (
genloader
.map(self.__resize_and_preprocess, num_parallel_calls=self.AUTO)
.batch(self.BATCH_SIZE, num_parallel_calls=self.AUTO)
.prefetch(self.AUTO)
)
self.genloader = genloader
# prepare embeddings
count = math.ceil(self.SAMPLE_SIZE/self.BATCH_SIZE)
## compute embeddings for real images
print("Computing Real Image Embeddings")
self.real_image_embeddings = self.__compute_embeddings(self.trainloader, count)
## compute embeddings for generated images
print("Computing Generated Image Embeddings")
self.generated_image_embeddings = self.__compute_embeddings(self.genloader, count)
# assert self.real_image_embeddings.shape == self.generated_image_embeddings.shape, "Embeddings are not of the same size"
print("Computed Embeddings\tReal Images Embedding Shape: {}\tGenerated Images Embedding Shape".format(
self.real_image_embeddings.shape,
self.generated_image_embeddings.shape
))
# method to produce evaluation results
@tf.autograph.experimental.do_not_convert
def evaluate(self):
# calculate Frechet Inception Distance
fid = self.__calculate_fid(self.real_image_embeddings, self.generated_image_embeddings)
print('The computed FID score is:', fid)
return fid
# method to generate embeddings from inception model
def __compute_embeddings(self, dataloader, count):
image_embeddings = []
for _ in tqdm(range(count)):
images = next(iter(dataloader))
embeddings = self.INCEPTION.predict(images)
image_embeddings.extend(embeddings)
return np.array(image_embeddings)
## STATIC METHODS: these methods knows nothing about the class
# static method to prepare the data before computing Inception Embeddings
@staticmethod
def __resize_and_preprocess(image):
# image *= 255.0 # original image are scaled to [0, 1], scaling back to [0, 255]
image = tf.image.convert_image_dtype(image, dtype=tf.float32, saturate=True)
# .preprocess_input() expects an image of scale [0, 255]
image = preprocess_input(image)
# inception model expects an image of shape (None, 299, 299, 3)
image = tf.image.resize(image, (299, 299), method='nearest')
return image
# static method to calculate frechet inception distance based on embeddings
@staticmethod
def __calculate_fid(real_embeddings, generated_embeddings):
# calculate mean and covariance statistics
mu1, sigma1 = real_embeddings.mean(axis=0), np.cov(real_embeddings, rowvar=False)
mu2, sigma2 = generated_embeddings.mean(axis=0), np.cov(generated_embeddings, rowvar=False)
# calculate sum squared difference between means
ssdiff = np.sum((mu1 - mu2)**2.0)
# calculate sqrt of product between cov
covmean = sqrtm(sigma1.dot(sigma2))
# check and correct imaginary numbers from sqrt
if np.iscomplexobj(covmean):
covmean = covmean.real
# calculate score
fid = ssdiff + np.trace(sigma1 + sigma2 - 2.0 * covmean)
return fid
%%time
fid_class = GAN_FID(batch_size=512, latent_dim=128, sample_size=10000, buffer_size=1024)
fid_class.fit(generator=gen4, train_data=test_images)
fid_score = fid_class.evaluate()
Computing Real Image Embeddings
0%| | 0/20 [00:00<?, ?it/s]
16/16 [==============================] - 3s 146ms/step
5%|▌ | 1/20 [00:03<01:13, 3.87s/it]
16/16 [==============================] - 2s 152ms/step
10%|█ | 2/20 [00:06<00:59, 3.30s/it]
16/16 [==============================] - 2s 99ms/step
15%|█▌ | 3/20 [00:08<00:47, 2.78s/it]
16/16 [==============================] - 1s 92ms/step
20%|██ | 4/20 [00:10<00:39, 2.47s/it]
16/16 [==============================] - 2s 133ms/step
25%|██▌ | 5/20 [00:13<00:37, 2.53s/it]
16/16 [==============================] - 3s 180ms/step
30%|███ | 6/20 [00:16<00:39, 2.80s/it]
16/16 [==============================] - 2s 109ms/step
35%|███▌ | 7/20 [00:19<00:34, 2.62s/it]
16/16 [==============================] - 1s 73ms/step
40%|████ | 8/20 [00:20<00:27, 2.33s/it]
16/16 [==============================] - 1s 89ms/step
45%|████▌ | 9/20 [00:22<00:24, 2.21s/it]
16/16 [==============================] - 2s 141ms/step
50%|█████ | 10/20 [00:25<00:24, 2.44s/it]
16/16 [==============================] - 2s 125ms/step
55%|█████▌ | 11/20 [00:28<00:22, 2.47s/it]
16/16 [==============================] - 2s 131ms/step
60%|██████ | 12/20 [00:30<00:20, 2.53s/it]
16/16 [==============================] - 2s 145ms/step
65%|██████▌ | 13/20 [00:33<00:18, 2.62s/it]
16/16 [==============================] - 2s 123ms/step
70%|███████ | 14/20 [00:36<00:15, 2.59s/it]
16/16 [==============================] - 2s 129ms/step
75%|███████▌ | 15/20 [00:38<00:12, 2.59s/it]
16/16 [==============================] - 2s 142ms/step
80%|████████ | 16/20 [00:41<00:10, 2.65s/it]
16/16 [==============================] - 2s 143ms/step
85%|████████▌ | 17/20 [00:44<00:08, 2.74s/it]
16/16 [==============================] - 2s 147ms/step
90%|█████████ | 18/20 [00:47<00:05, 2.93s/it]
16/16 [==============================] - 3s 176ms/step
95%|█████████▌| 19/20 [00:51<00:03, 3.06s/it]
16/16 [==============================] - 2s 151ms/step
100%|██████████| 20/20 [00:54<00:00, 2.72s/it]
Computing Generated Image Embeddings
0%| | 0/20 [00:00<?, ?it/s]
16/16 [==============================] - 2s 128ms/step
5%|▌ | 1/20 [00:02<00:48, 2.56s/it]
16/16 [==============================] - 2s 144ms/step
10%|█ | 2/20 [00:05<00:48, 2.71s/it]
16/16 [==============================] - 2s 126ms/step
15%|█▌ | 3/20 [00:07<00:44, 2.64s/it]
16/16 [==============================] - 2s 155ms/step
20%|██ | 4/20 [00:10<00:44, 2.80s/it]
16/16 [==============================] - 2s 146ms/step
25%|██▌ | 5/20 [00:13<00:42, 2.84s/it]
16/16 [==============================] - 3s 178ms/step
30%|███ | 6/20 [00:17<00:42, 3.07s/it]
16/16 [==============================] - 4s 236ms/step
35%|███▌ | 7/20 [00:21<00:44, 3.45s/it]
16/16 [==============================] - 2s 156ms/step
40%|████ | 8/20 [00:24<00:40, 3.35s/it]
16/16 [==============================] - 3s 184ms/step
45%|████▌ | 9/20 [00:28<00:38, 3.53s/it]
16/16 [==============================] - 2s 129ms/step
50%|█████ | 10/20 [00:31<00:32, 3.24s/it]
16/16 [==============================] - 7s 492ms/step
55%|█████▌ | 11/20 [00:39<00:42, 4.70s/it]
16/16 [==============================] - 8s 526ms/step
60%|██████ | 12/20 [00:48<00:48, 6.08s/it]
16/16 [==============================] - 3s 199ms/step
65%|██████▌ | 13/20 [00:52<00:37, 5.36s/it]
16/16 [==============================] - 4s 236ms/step
70%|███████ | 14/20 [00:56<00:30, 5.13s/it]
16/16 [==============================] - 3s 167ms/step
75%|███████▌ | 15/20 [01:00<00:22, 4.56s/it]
16/16 [==============================] - 4s 242ms/step
80%|████████ | 16/20 [01:04<00:17, 4.49s/it]
16/16 [==============================] - 2s 140ms/step
85%|████████▌ | 17/20 [01:07<00:11, 4.00s/it]
16/16 [==============================] - 2s 135ms/step
90%|█████████ | 18/20 [01:09<00:07, 3.63s/it]
16/16 [==============================] - 2s 152ms/step
95%|█████████▌| 19/20 [01:13<00:03, 3.61s/it]
16/16 [==============================] - 7s 445ms/step
100%|██████████| 20/20 [01:22<00:00, 4.11s/it]
Computed Embeddings Real Images Embedding Shape: (10240, 2048) Generated Images Embedding Shape The computed FID score is: 5.453099102467402 CPU times: total: 4min 46s Wall time: 2min 24s
WGAN is a better GAN than GAN because it has a better loss function and it is more stable. However, the images are not superb and I would say the images are kind of the same or maybe just slightly better than the original DCGAN images. I would say it is too computationally expensive to continue exploring WGAN and even trying out Data augmentation which will take a lot longer than the current Spectral Normalization CGAN that i did before this. Therefore I will submit the images generated by Spectral Normalization CGAN as my final submission.